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EVALUATIONS  OF  GLOBAL  WIND  PREDICTION  AT 
FLEET  NUMERICAL  METEOROLOGY  AND  OCEANOGRAPHY 
CENTER:  FROM  THE  PERSPECTIVE  OF  A  WAVE  MODELER 


Abstract 

This  study  describes  the  validation  of  winds  fields  used  in  a  companion  study 
(“Evaluations  of  global  wave  prediction  at  Fleet  Numerical  Meteorology  and 
Oceanography  Center”,  submitted  for  review).  During  this  validation,  it  is  found  that 
conventional  methods  of  estimating  the  distribution  of  bias  across  wind  speeds  can 
produce  widely  misleading  conclusions.  This  is  demonstrated,  and  a  simple  solution 
(histogram  comparisons  without  geographic  interpolation)  is  presented. 

Introduction 

The  NOGAPS  (Hogan  and  Rosmond  1991)  wind  fields,  NOGAPS/QuikSCAT 
wind  fields,  and  the  wave  model  hindcasts  are  introduced  in  the  paper  “Evaluations  of 
global  wave  prediction  at  Fleet  Numerical  Meteorology  and  Oceanography  Center”, 
submitted  for  review.  Related  manuscripts  are  included  in  the  References  section  of  this 
report. 

Wind  field  Validation 

1  Ground  Truth 

The  wind  fields  used  to  force  the  hindcasts  are  compared  to  in  situ  wind  data. 
National  Data  Buoy  Center  (NDBC)  buoys  are  selected  from  the  general  population  of 
NDBC  buoys  with  preference  given  to  buoys  further  offshore,  as  these  are  expected  to  be 
more  representative  of  the  open  ocean,  where  the  vast  majority  of  wave  energy  is 
generated.  Seven  buoys  each  in  the  Pacific  Ocean  (46066,  46035,  46001,  46005,  46002, 
46006,  46059)  and  Atlantic  Ocean  (4401 1,  44005,  44018,  44008,  44004,  41001,  41002) 
were  used.  Their  locations  are  shown  in  Fig.  1 .  Also  shown  in  Fig.  1  is  the  spatial 
distribution  of  the  fractions  of  occurrence  of  gale  force  winds  ( Ul0  >  1 5  m/s  )  during  the 

winter  2002/2003  hindcast  period  (in  the  blended  NOGAPS/QuikSCAT  wind  fields, 
tabulated  at  three  hour  intervals)  in  the  north  Pacific  and  north  Atlantic.  The  areas  with 
high  occurrence  of  strong  winds  may  be  considered  the  predominant  swell  generation 
regions  during  the  period.  It  is  apparent  from  the  figure  that  the  buoy  network  is 
reasonable — though  far  from  ideal — for  validating  winds  in  the  predominant  swell 
generation  regions:  the  buoys  are  located  in  moderately  active  areas.  In  this  validation, 
buoy  wind  speeds  are  converted  to  10  m  elevation  assuming  neutral  stratification.  Except 
where  stated  otherwise,  model  forcing  fields  are  collocated  with  the  buoys  via  tri-linear 
interpolation  (two  space  coordinates  plus  time  coordinate). 

In  the  validations  to  follow,  the  accuracy  of  our  “ground  truth”  is  an  important 
consideration.  Hamilton  (1980)  and  Gilhousen  (1987)  report  typical  RMS  (thus  not 
necessarily  systematic  bias)  wind  speed  errors  (in  buoy  data)  as  being  1.0  ms'1  or  better. 
Yet  it  is  expected  that  buoy  wind  speed  measurements  are  less  accurate  in  high  winds  due 
to  the  movement  of  the  buoys  in  corresponding  high  seas  (Ebuchi  et  al.  2002),  as  well  as 
wind  profile  distortion  by  large  waves  (Large  et  al.  1995);  only  very  limited  validation  of 
buoy  anemometer  measurements  against  fixed  platform  measurements  in  high  winds 
have  been  performed.  Despite  these  shortcomings,  the  buoy-mounted  anemometer 

Manuscript  approved  September  7,  2004. 


Rogers  et.  al 
11:12  AM  11/5/2004 


Wind  field  validation 


1/15 


measurements  are  our  best  option  for  ground-truth,  as  they  are  the  most  direct  open  ocean 
wind  measurements  available  to  us. 

Data  were  not  available  from  all  buoys  during  all  time  periods  of  interest.  Where 
data  did  exist,  they  were  compared  to  corresponding  values  in  the  model  forcing  fields. 
The  wind  fields  were  interpolated  to  the  buoy  locations  as  appropriate.  For  each  forcing 
type  and  hindcast  period  (four  combinations  total),  all  collocated  points  were  combined 
into  a  conventional  scatter-plot  comparison  (these  plots  can  be  obtained  from  the  author; 
they  are  not  included  here). 

2  Results 

Validation  comparisons  were  created  for  both  hindcast  periods  and  for  three  wind 
forcing  field  types:  1)  NOGAPS,  2)  NOGAPS  blended  with  QuikSCAT  measurements 
(quality  control  flagged  QuikSCAT  measurements  included),  and  3)  NOGAPS  blended 
with  QuikSCAT  measurements  (quality  control  flagged  QuikSCAT  measurements 
omitted).  The  latter  are  referred  to  as  “filtered”  NOGAPS/QuikSCAT  fields  below.  Both 
“filtered”  and  “unfiltered”  fields  were  included  in  the  validation  to  quantify  the  impact  of 
including  flagged  measurements  and  to  subsequently  decide  which  to  use  in  the  wave 
model  hindcasts. 

The  traditional  method  of  wind  field  validation  is  to  present  scatter  plot 
comparisons  of  measurements  against  collocated  wind  field  values,  along  with  statistics 
such  as  RMS  error,  bias,  and  regression  parameters.  The  RMS  error  and  bias 
unfortunately  are  of  limited  value  to  this  study,  since  they  do  not  provide  information 
regarding  the  error  at  different  wind  speeds,  which  is  crucial  here  because  of  the  obvious 
increased  sensitivity  of  a  wave  model  to  higher  wind  speeds.  The  regression  parameters 
do  contain  this  sort  of  information,  but  as  will  be  demonstrated  below,  various  factors  can 
lead  to  widely  misleading  regressions.  Much  of  the  focus  of  this  validation  will  be  on 
addressing  these  problems. 

Due  to  space  limitations,  we  cannot  present  all  results  here.  Figure  2(a,b)  shows 
some  of  the  results  from  the  validation  for  the  2001/2002  filtered,  blended 
NOGAPS/QuikSCAT  forcing  fields.  The  apparent  positive  bias  at  low  wind  speeds  is 
probably  due  to  the  nature  of  the  comparison:  it  is  well  known  that  comparison  of  scalar 
wind  speeds  tends  to  suggest  spurious  relations  near  the  origin  of  a  scatter  plot  (i.e.  at  low 
wind  speeds);  see  e.g.  Freilich  (1997),  Dickenson  et  al.  (2001),  Ebuchi  et  al.  (2002).  Such 
spurious  relations  will  obviously  tend  to  skew  linear  regressions  of  scatter  plots.  These 
two  plots  suggest  that  at  moderate  to  high  wind  speeds  (5-22  ms'1),  the  mean  error  (i.e. 
bias)  of  the  wind  field  is  relatively  slight  (less  than  1  ms'1)  and  is  negative  at  higher  wind 
speeds. 

During  both  time  intervals,  the  RMS  error — calculated  from  the  entire  wind  speed 
range — of  the  filtered  NOGAPS/QuikSCAT  fields  is  greater  than  the  RMS  error  of  the 
NOGAPS  fields:  2.12  ms'1  vs.  1.96  ms'1  in  the  2001/2002  period  and  2.17  ms'1  vs.  1.99 
ms'  during  the  2002/2003  period.  This  is  not  surprising,  since  the  atmospheric  model  is 
expected  to  be  fairly  accurate  predicting  low  and  moderate  wind  speeds,  whereas  the 
process  of  blending  in  QuikSCAT  measurements  introduces  sampling  error  (e.g.  Schlax 
et  al.  2001),  which  tends  to  be  random. 

The  comparison  method  of  Fig.  2  (scatter  plots  and  binning  by  buoy  wind  speed) 
is  fairly  mainstream,  having  been  used  by  others  such  as  Ebuchi  et  al.  (2002),  yet  it  is 
possible  to  make  some  additional  comparison  more  pertinent  to  wave  model  sensitivity. 
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Focusing  on  the  mean  error  of  Fig  2b,  it  can  be  argued  that  the  practical  impact  of  a  wind 
speed  “bin"  should  be  weighted  according  to  the  population  of  that  bin.  Further  the 
disproportionate  impact  that  high  wind  speeds  have  on  wave  climate  should  also  be 
recognized.  We  can  do  this  by  weighting  each  wind  speed  bin  according  to  the  wave 
energy  that  it  might  produce.  For  this  purpose,  we  use  the  wave  height  calculated  from 
the  so-called  “Pierson-Moskowitz  (PM)  spectrum”  (Pierson  and  Moskowitz  1 964) 1 .  This 
can  be  interpreted  as  a  weighting  according  to  the  square  of  the  wind  speed.  The  mean 
errors  for  the  three  winter  2001/2002  forcing  fields,  weighted  in  this  manner,  are  shown 
in  Figure  3.  This  comparison  suggests  that  overall  all  bias  is  smallest  in  the  NOGAPS 
fields  blended  with  unfiltered  QuikSCAT  data. 

There  is  another  qualification  that  must  be  mentioned.  During  the  validation 
process,  it  was  noticed  that  bias  comparisons  such  as  Fig.  3  could  be  misleading  at  the 
higher  wind  speeds.  This  effect  has,  in  fact,  been  noticed  by  others.  Freilich  and  Dunbar 
(1999)  note  that  apparent  negative  bias  at  high  wind  speeds  may  “be  the  result  of  binning 
on  buoy  wind  speeds.... Wentz  et  al.  (1984)  and  Chelton  and  Wentz  (1986)  showed  that 
apparent  underpredictions  in  high  wind  scatter  plots  were  expected  if  the  buoy 
measurements  had  random  errors.”  [The  topic  is  also  discussed  by  Tolman  (1998).]  Note 
that  this  spurious  effect  will  tend  to  have  a  similar  effect  on  the  slope  of  scatter  plot 
regressions  as  does  the  low  wind  speed  problem  mentioned  above.  Acknowledging  that 
this  effect  exists,  it  is  reasonable  to  speculate  that  the  winter  2001/2002  blended 
NOGAPS/QuikSCAT  wind  forcing  field  may  actually  have  a  slight  positive  bias  at  high 
wind  speeds  rather  than  the  slight  negative  bias  apparent  here,  and  the  negative  bias  in  the 
2001/2002  NOGAPS  wind  forcing  field  at  high  wind  speeds,  though  significant,  is 
probably  exaggerated  in  this  comparison. 

Fortunately,  we  have  another  way  to  present  the  collocation  comparisons  which 
does  not  share  this  problem:  to  create  a  histogram  of  all  buoy  measurements  and  compare 
to  a  histogram  of  the  collocated  wind  forcing  field  values.  Though  this  does  not  give  any 
indication  of  random  errors  (e.g.  errors  resulting  from  a  misplaced  cyclone),  it  does  tell 
us  much  about  systematic  bias  (e.g.  errors  from  consistently  underpredicted  cyclone 
strength),  and  random  error  in  the  buoy  data  should  not  affect  our  interpretation.  Figure  4 
shows  such  a  comparison  for  the  winter  2001/2002  period,  with  the  number  of 
occurrences  weighted  by  the  PM  wave  height,  as  in  Fig.  3.  This  comparison  (Fig.  4)  tends 
to  favor  the  blended  NOGAPS/QuikSCAT  wind  fields  created  with  filtered  QuikSCAT 
measurements.  The  unfiltered  NOGAPS/QuikSCAT  fields  show  a  positive  bias  at  the 
higher  wind  speeds. 

To  summarize,  the  merits  (with  regard  to  wind  speed  bias)  of  using  or  omitting 
quality-control  flagged  QuikSCAT  data  depends  on  the  method  of  validation.  If  we 
assume  that  random  error  in  the  buoy  data  is  negligible2,  we  may  bin  wind  speed 
according  to  buoy  wind  speeds.  This  comparison  supports  using  flagged  data  (Fig.  3).  On 


1  Though  lacking  the  sophistication  of  a  full  wave  model,  the  PM  spectrum  suits  our  purposes  because  it  is 
simple  and  widely  known.  The  PM  spectrum  gives  us  a  simple  relation  between  wind  speed  and  wave 
height  under  a  number  of  large  assumptions,  one  being  that  the  local  wave  condition  is  “fully  developed”. 
This  implies  that  the  wave  state  is  as  it  would  be  if  the  wind  has  been  blowing  at  the  same  speed  and 
direction  for  an  infinite  duration  over  an  infinite  fetch.  Here,  we  use  the  form  of  the  PM  spectrum  given  by 
Komen  et  al.  (1984). 

'  Buoy  wind  speeds  are  typically  calculated  on  8-minute  averages.  This  would  seem  to  support  the 
argument  that  random  error  is  small. 
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the  other  hand,  histogram  comparisons  support  omitting  flagged  data  (Fig.  4).  Since  the 
latter  is  much  less  sensitive  to  random  error,  we  choose  to  omit  flagged  data. 

Thus,  the  unfiltered  fields  are  dismissed  as  a  candidate  for  hindcast  forcing  and  are  not 
included  in  the  additional  validation  comparisons  below. 

Yet,  that  is  not  the  end  of  the  story!  The  act  of  geographic  interpolation  to 
collocate  forcing  fields  with  buoy  location  is  expected  to  have  a  smoothing  effect  in  cases 
where  the  signals  (e.g.  those  due  to  cold  fronts)  in  a  field  of  10  m  wind  speeds  are  not 
well  described  by  a  model  grid3.  Again,  the  histogram  approach  gives  us  an  advantage: 
slight,  random  shifts  in  individual  buoy  locations  are  not  expected  to  have  a  marked 
effect  on  the  histogram  created  from  all  14  buoys.  Thus,  we  can  skip  interpolation  by 
simply  taking  model  output  at  the  model  node  nearest  each  buoy  and  considering  it 
collocated  (i.e.  nearest  neighbor  collocation).  This  experiment  yields  histograms  of 
different  shape  (for  example,  with  nearest  neighbor  comparison,  note  the  positive  bias 
that  is  more  pronounced  in  the  filtered  NOGAPS/QuikSCAT  fields  in  the  15-20ms'‘ 
range).  Comparison  via  the  nearest  neighbor  approach  is  expected  to  be  less  sensitive  to 
spurious  effect. 

Figure  5(a,b)  shows  the  results  for  the  2001/2002  and  2002/2003  periods.  These 
results  can  be  distilled  further  by  considering  the  mismatch  (forcing  field  histogram  vs. 
buoy  histogram)  in  individual  wind  speed  bins  (Table  1).  With  the  histogram  mismatch  as 
a  metric  of  forcing  field  bias,  these  comparisons  suggest  the  following: 

1)  Of  the  four  forcing  fields,  the  2001/2002  NOGAPS  has  the  most  severe  bias. 

2)  The  2002/2003  NOGAPS  field  has  the  smallest  bias. 

3)  The  apparent  bias  in  the  NOGAPS  fields  (for  both  time  periods)  is  expected  to 
result  in  an  underprediction  of  wave  energy  by  a  “perfect”  wave  model.  The 
negative  bias  is  primarily  at  10m  wind  speeds  greater  than  12  ms'1. 

4)  The  apparent  bias  in  the  blended  NOGAPS/QuikSCAT  fields  is  expected  to  result 
in  an  overprediction  of  wave  energy  by  a  “perfect”  wave  model.  The  positive  bias 
is  primarily  at  10m  wind  speeds  greater  than  15  ms'1. 

5)  The  apparent  bias  in  the  blended  NOGAPS/QuikSCAT  fields  is  remarkably 
similar  for  the  two  time  periods.  This  might  be  taken  as  an  indication  of  the 
robustness  of  the  method. 

Note  that  we  do  not  know  from  this  comparison  how  much  of  the  error  in  the 
NOGAPS/QuikSCAT  wind  fields  is  from  NOGAPS  and  how  much  is  from  the 
QuikSCAT  measurements.  However,  the  literature  gives  us  some  insight  into  this.  The 
mission  requirements  for  QuikSCAT  are  to  achieve  accuracy  within  ±2  ms'1  for 
3  ms'1  <  Uw  <  20  ms'1  and  ±10%  for  20  ms'1  <  t/10  <  30  ms'1  (PODAAC  2001).  Ebuchi  et 

al.  (2002)  evaluated  the  accuracy  of  the  L2B  QuikSCAT  data  (with  data  flagged  for 
quality  control  removed)  using  a  large  number  of  offshore  buoy  measurements  as  ground 
truth.  They  concluded  that  the  mission  (i.e.  accuracy)  requirements  are  satisfied.  Using 
comparisons  similar  to  Fig.  2,  they  find  no  significant  dependency  on  wind  speed,  except 
at  high  wind  speeds  ( Ui0  >  15  ms’1 ),  where  there  is  a  slight  positive  bias. 


3  Imagine  the  extreme  case  where  the  time  series  at  one  node  in  a  one-dimensional  model  is  a  sine  wave, 
and  the  time  series  at  a  neighboring  node  is  the  same  signal,  except  with  a  phase  shift  (say,  a  sine  wave 
offset  by  half  a  cycle).  Interpolation  to  an  intermediate  location  will  yield  a  damped  time  series. 
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Conclusions 


Using  direct  validation  of  NOGAPS  analyses  against  buoy  data,  it  is  shown  herein 
that  the  negative  bias  at  high  wind  speeds  is  much  reduced  during  the  winter  of 
2002/2003,  compared  to  the  previous  winter. 

During  the  validation  of  wind  fields,  it  is  observed  that  traditional  metrics  such  as 
wind  speed  bias  and  RMS  error  are  of  limited  value  in  the  context  of  wave  modeling, 
since  they  do  not  provide  information  regarding  the  error  at  different  wind  speeds,  which 
is  crucial  because  of  the  obvious  increased  sensitivity  of  a  wave  model  to  higher  wind 
speeds.  Further,  traditional  methods  of  validation  (e.g.  scatter  plot  diagrams)  produce 
apparent  bias  at  low  and  high  wind  speeds  which  are  likely  spurious.  For  evaluation  of 
wind  speed  bias,  this  can  be  circumvented  by  comparing  histograms  of  collocated  wind 
speeds.  The  method  of  collocation  also  is  shown  to  have  a  strong  influence  on  the 
apparent  bias.  Further,  we  show  how  a  simple  modification  of  validation  plots — 
weighting  according  to  wind  speed — allow  us  to  better  quantify  the  expected  impact  of 
wind  speed  bias  on  wave  model  results. 

Acknowledgements 

The  authors  thank  the  FNMOC  Satellite  Data  Team,  Drs.  Hendrik  Tolman 
(NCEP/SAIC),  and  Keith  Sashegyi  (NRL  Marine  Meteorology  Division)  for  providing 
varied  discussions,  information,  and  resources.  This  work  was  funded  by  the  Office  of 
Naval  Research  Ocean  Modeling  Program.  The  earlier  work  which  formed  the  foundation 
of  the  present  study  was  supported  by  the  Naval  Research  Laboratory  Core  Program 
under  Program  Element  62435N  (Principle  Investigator:  Dr.  James  Kaihatu). 


Rogers  et.  al 
11:12  AM  11/5/2004 


Wind  field  validation 


5/15 


Table  1.  Statistics  derived  from  wind  field  validation.  The  first  statistic  is  simple  RMS 
error  from  the  scatter  plot  of  Fig.  2.  The  other  two  statistics  are  from  the  histograms  of 
Fig.  5  (histograms  of  the  buoy  measurements  and  collocated  wind  forcing  field  values, 
weighted  by  population  and  Pierson-Moskowitz  wave  height,  collocated  via  nearest 
neighbor  method).  The  two  forcing  fields  are  NOGAPS  fields  and  combined 
NOGAPS/QuikSCAT  fields  (flagged  QuikSCAT  measurements  filtered  out).  Here,  A,  is 

the  buoy/model  mismatch  in  the  histogram  at  an  individual  wind  speed  bin  Since  the 
population  of  collocated  point  differ  in  the  two  hindcasts  (due  to  availability  of  buoy 
data),  these  results  have  been  normalized  by  the  total  population. 


Hindcast 

Field 

RMS  error 

(ms  ) 

n 

(m) 

/=1 

ZK|  (m) 

/=1 

2001/2002 

NOGAPS 

1.96 

-0.205 

0.490 

NOGAPS/QuikSCAT 

2.12 

+0.172 

0.208 

2002/2003 

NOGAPS 

1 . 99 

-0.060 

0.193 

NOGAPS/QuikSCAT 

2.17 

+0.169 

0.219 
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Figure  1 .  Locations  of  buoys  used  in  comparisons  to  wind  fields  used  as  wave  model 
input  are  shown.  Color  shading  indicates  the  fraction  of  occurrence  of  gale  force  winds 
( Uw  >15  m/s )  during  the  winter  2002/2003  hindcast  period  (in  the  blended 

NOGAPS/QuikSCAT  wind  fields).  Areas  with  high  fraction  of  occurrence  can  be 
considered  the  primary  regions  of  swell  generation  in  during  the  period. 
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Figure  2a.  Validation  plot  for  the  filtered,  blended  NOGAPS/QuikSCAT  forcing  wind 
field  for  the  2001/2002  hindcast.  Color  shading  indicates  the  number  of  occurrences  in  1 
ms'1  x  1  ms'1  bins.  The  horizontal  axis  is  the  buoy  10  m  wind  speed.  The  vertical  axis  is 
the  residual  wind  speed,  calculated  as  the  difference  between  the  collocated  forcing  field 
wind  speed  and  the  buoy  wind  speed.  This  comparison — and  all  other  wind  field 
comparisons  herein — are  obtained  using  collocations  with  data  from  14  buoys. 
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Figure  2b.  Validation  plot  for  the  filtered,  blended  NOGAPS/QuikSCAT  forcing  wind 
field  for  the  2001/2002  hindcast.  Individual  errors,  calculated  as  the  difference  between 
collocated  buoy  measurements  and  wind  forcing  field  values,  have  been  binned  according 
to  the  buoy  10  wind  speed  (in  1  ms'1  bins).  The  mean  error  for  each  bin  is  indicated  by 
the  non-vertical  line  and  the  standard  deviations  are  indicated  with  vertical  lines  (±1 
standard  deviation). 
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2001 .2002 


Figure  3.  Mean  error  for  the  winter  2001/2002  wind  fields,  binned  according  to  buoy 
wind  speed  and  weighted  by  population  and  Pierson-Moskowitz  wave  height  (see  text). 
The  three  forcing  fields  shown  are:  NOGAPS  fields,  combined  NOGAPS/QuikSCAT 
fields  (flagged  QuikSCAT  measurements  filtered  out),  combined  NOGAPS/QuikSCAT 
fields  (unfiltered). 
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2001 .2002 


Figure  4.  Histograms  for  the  buoy  measurements  and  collocated  wind  forcing  field 
values,  weighted  by  population  and  Pierson-Moskowitz  wave  height  (see  text).  The  three 
forcing  fields  shown  are:  NOGAPS  fields,  combined  NOGAPS/QuikSCAT  fields 
(flagged  QuikSCAT  measurements  filtered  out),  combined  NOGAPS/QuikSCAT  fields 
(unfiltered).  The  winter  2001/2002  result  is  shown  here.  In  this  comparison,  wind  field 
values  have  been  determined  at  buoy  locations  using  bilinear  interpolation. 
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Fig.  5.  Histograms  for  the  buoy  measurements  and  collocated  wind  forcing  field  values, 
weighted  by  population  and  Pierson-Moskowitz  wave  height  (see  text).  The  two  forcing 
fields  shown  are  NOGAPS  fields  and  combined  NOGAPS/QuikSCAT  fields  (flagged 
QuikSCAT  measurements  filtered  out).  In  this  comparison,  wind  field  values  have  been 
determined  at  buoy  locations  by  using  the  value  in  the  nearest  model  node  {not  bilinear 
interpolation,  as  was  used  in  Fig.  5). 

Fig  5a.  Winter  2001/2002  result 
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2002.2003 


Fig  5b.  Winter  2002/2003  result 
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